Prompt tuning (PT) has long been recognized as an effective and efficient paradigm for transferring large pretrained vision-language models (VLMs) to downstream tasks by learning a tiny set of context vectors. Nevertheless, in this work, we reveal that freezing the parameters of VLMs during learning the context vectors neither facilitates the transferability of pre-trained knowledge nor improves the memory and time efficiency significantly. Upon further investigation, we find that reducing both the length and width of the feature-gradient propagation flows of the full fine-tuning (FT) baseline is key to achieving effective and efficient knowledge transfer. Motivated by this, we propose Skip Tuning, a novel paradigm for adapting VLMs to downstream tasks. Unlike existing PT or adapter-based methods, Skip Tuning applies Layer-wise Skipping (LSkip) and Class-wise Skipping (CSkip) upon the FT baseline without introducing extra context vectors or adapter modules. Extensive experiments across a wide spectrum of benchmarks demonstrate the superior effectiveness and efficiency of our Skip Tuning over both PT and adapter-based methods. Code: https://github.com/anonymity-007/SkipT.
Paperid:2189
Authors:Ashmal Vayani , Dinura Dissanayake , Hasindri Watawana , Noor Ahsan , Nevasini Sasikumar , Omkar Thawakar , Henok Biadglign Ademtew , Yahya Hmaiti , Amandeep Kumar , Kartik Kuckreja , Mykola Maslych , Wafa Al Ghallabi , Mihail Minkov Mihaylov , Chao Qin , Abdelrahman Shaker , Mike Zhang , Mahardika Krisna Ihsani , Amiel Gian Esplana , Monil Gokani , Shachar Mirkin , Harsh Singh , Ashay Srivastava , Endre Hamerlik , Fathinah Asma Izzati , Fadillah Adamsyah Maani , Sebastian Cavada , Jenny Chim , Rohit Gupta , Sanjay Manjunath , Kamila Zhumakhanova , Feno Heriniaina Rabevohitra , Azril Hafizi Amirudin , Muhammad Ridzuan , Daniya Najiha Abdul Kareem , Ketan Pravin More , Kunyang Li , Pramesh Shakya , Muhammad Saad , Amirpouya Ghasemaghaei , Amirbek Djanibekov , Dilshod Azizov , Branislava Jankovic , Naman Bhatia , Alvaro Cabrera Berobide , Johan Obando-Ceron , Olympiah Otieno , Fabian Farestam , Muztoba Rabbani , Sanoojan Baliah , Santosh Sanjeev , Abduragim Shtanchaev , Maheen Fatima , Thao Nguyen , Amrin Kareem , Toluwani Aremu , Nathan Augusto Zacarias Xavier , Amit Bhatkal , Hawau Olamide Toyin , Aman Chadha , Hisham Cholakkal , Rao Anwer , Michael Felsberg , Jorma Laaksonen , Thamar Solorio , Monojit Choudhury , Ivan Laptev , Mubarak Shah , Salman Khan , Fahad Shahbaz Khan